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ABSYBACT 

In the first aection of the docuaent the author uoea 
reviews of selected studies to provide a cross-^section of 
aethodologies used in evaluating vocational and technical education 
prograas. The second section taken a fotaal aodel developed foe the 
evaluation of vocational education (Central Kentucky Beoearch 
Coordinating Unit, 1972) and explains the types of infocaation and 
■ethods which can bo us^d to obtain the inforaation at each phasQ o£ 
the aodel. The last section of the document provides suggestions 
derived fros the author^s experience vhile coordinating the 
evaluation of the Skyline Career Devulopaent Center in Dallas, Texas. 
Thes«i coL»iderations arc broken down into context and process 
evaluation. An epilogue, flow charts, and a S4*itca bibliography are 
included. (For coapanion docuaonts covering Facilities Evaluation, 
Personnel Evaluation, and Student evaluation see CE 000 968, CE 001 
153 respectively.) (Author/KP) 
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FOREWORD 



Vocaiional A\ui trclinical cdiicarton has enjoyed liigli viisibility during the past few years 
and with it increased pressure in account for expenditure.^ and to jl^stify programs. As a 
result, educators are ever alert for effective means of evaluating their educational pr(jgrams. 
This publication and its three companion documents (PcrwnncI livaluatioft in Wicationdi 
diul Tcchttictil liiiucatiou, Vacililies liviiluafiou in I 'ocational dful Technical lUhcatiou, and 
Stuiicfit lh\t(\uition in l^orutfUMii/ lUu/ Icclinidil l-AhiaMion) provide educational practitioners 
witli a review and synthesis of the most important works ni evaluation as it applies to voca- 
tional and technical education. 

In Vrojitam Hvaluation in Vocational and Technical l-Alucntion, the author previews 
selected studies which provide a cross section of methodology used in evaluating vocational 
and technical education programs. The various phases of program evaluation arc documented 
and context evaluation and process evaluation arc addressed in terms of a real situation at the 
Skyline Career Development Center in Dallas. 

The profession is indebted lo William T. Denton (or his scholarship in the preparation 
of this report. Recognition is also due Gordon Law. Department of Urban Education, Rutgers- 
the State University; and Donald L. Rathbun, Associate Director, American Vocational Asso- 
ciation for their critical review of the manuscript prior to final revision and publication. Paul 
E. Schroeder coordinated the publication's development, ,ind Alice J. Brown and Paula Kurth 
provided ihe technical c'liting. 

Robert E. Taylor 
Director 

The Center for Vocational and 
Technical Education 

ERIC Clearinghouse on Vocational 
and Technical Education 
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INTRODUCTION 



The purpose of this paper is to present inforiiintioii relative to the slate of the art in 
evaluation of vocational and technical education programs. The first section of this paper is 
not intended to be .ui exhaustive revi»;w of the literature; however, it docs present sonic reviews 
of studies in the field of vocational and technical education that are related to the topic: 
evaluation of programs. These studies were chosen for review so the reader might ohtuin ;i 
feeling for the wide range of methods by which programs are evaluated. The results section 
of the reviews demonstrates the diversity of information that can be provided through evalua- 
tion. 

The second section uf the paper takes a formal model devnoped for the evaluation of 
vocational education (Central Kentucky Research Coordinating Unit, 1972) and explains the 
types ol information and methods which can be used to obtain he information at eacli phase 
of the model. Appropriate references are made to the literature to direct the reader to addi- 
tional sources of inforniation. 

The third section of the paper is devcted to some suggestions which the author has found 
to be practical from his experience while coordinating the evaluation of the Skyline Career 
Development Center in Dallas, Texas. 

The anticipated audience for the paper is the practicing educator who finds himseU. 
because of cither external demands or personal desires, needing to evaluate a program \v r! ! 
area of vocational and technical education. The paper should also prove useful to the ad»« i 
istrator who wants to learn more about the complex field of evaluation. 

It is the position of the author that, although the demand for program evaluatio'^ is o.cen 
from an external funding source, the primary beneficiary of an evaluation is the local ^\cvi.v. 
That is, the information provided by a comprehensive evaluation for the purpose of rcruct jr- 
ing a program at the local level should exceed any requirements of an external agenc y. 1 h.o 
bias undoubtedly is reflected in this paper. 

For the sake of clarification, the definition of evaluation as used in this paper is that i f 
Cuba (1968:23) **Evaluation is the process of . . . obtaining and . . . providing . . . useful . . . 
information for making , . . educational decisions/* 
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A RISVIEW OF SELECTED STUDIES 



The following reviews «)f projects arc chosen to exciiiplify the diversity of ways in which 
evaliia ion itifortnatioti can lie obtained. The n;cthods of evaluation vary from statistical 
analysis to a review of documents. The personnel involved in collecting the information also 
vary from outside consultants to students. The important point in each case is that informa- 
tion was provided to effect t ;ie deciiion'niaking process. 

Educational agcticies arc always on the alert for new, innovative programs and the local 
evaluator is often asked to provide inputs about the relative merits of innovative programs 
prior to pilot testing. 

The Classification and Evaluation of Innovations in 
Vocational and Technical Eciucatio.i 

The following study was designed to develop a model for evaluating innovations.^ 

Project Overview . The project was set up to develop and field test a taxonomy of inno- 
vations appropriate for vocational and technical education. A consequent purpose was to 
develop guidelines for evaluating innovation characteristics. 

Purpose of the Evaluation . Once designed, the classification system was field tested to 
determine how project personnel at exemplary project sites would rate individual items. The 
input thus provided helped determine the final system. 

Methodology . The Innovations Evaluation Guide was initially developed after a review 
of the literature and interviews with six **innovative** superintendents and their staff. The 
guide was then pilot tested in ajunior high school for clarity and readability, followed by a 



^William L. Hull and Randall L, Wells, The Classification and Evaluation of Innovations 
in Vocational and Technical Educationy Final Report, Research and Development Series 
No. 71 (Columbus: The Center for Vocational and Technical Education, The Ohio State 
University, 1972). 



field ic%{ At lour cxmipljiry prnjrrf 5iitc5i which were chmen hy a hr;iiKli nf the U.S. Office 
of Education. The field te^it consisted of %{a(( inemhers at the cxeinpli^ry sites rating ei^ch 
of the items in the InuoVcitions Evaluation Ciuide as to whom the item wa% important: teacher, 
administrator, or both. Each item wa^ rated as to how essential it was. State supervisors and 
local project directors of exemplary programs ;^ttending a national conference aUf> participated 
in the field test. The data thus collected in the field test were used to make final revisions 
of the Innovations Evaluation Gmde. 

Results. comprehensive taxononiy which classifies discrete vocational technical 
innovations into mutually exclusive categories was not possible with our present level of 
knowledge and technology" (Hull and Wells, 1972:.^9). 

Probably i)f more importance, however, is that the Innovations Evaluation (luide can be 
usetl by those faced with the enormous problem of finding and choosing innovations to meet 
locally identified needs. It is interesting to note that in evaluating innovations, respondents 
chose as t!;c most essential items quantity of staff costs, availability costs, availability of 
dollars, space (bousing) :iources of dollars, hardware, and complexity. Those items chosen 
as least essential were; Mte of learning, entry and ad»^anccmcnt in an occupation, new rcia 
tionships among groups, cyclical considerations, economic and social efficiencies, reliability, 
and divisibility (Hull, 1972). 

The Annual Vocational Report of Louisiana^ Vocational 
and Technical Program -Fiscal Year 1972 

Oftenrimcs an evaluation nmst be done from existing documents, especially in the case 
of state agencies. This study is an example of such an evaluation.^' 

Project Overview . Any state desiring to receive funds under the Vocational Education 
Amendments of 1968, Public Law 90-576, nmst establish a state advisory council. Among 
other duties, the state advisory council must . . . '^evaluate vocational education programs, 
services and activities under this title, and publish and distribute the results thereof; . 
(Strong, 1972:1). This report fulfills that requirement. 



^Merle E. Strong and Daniel Jarosik, The Auuual Vocatioml Report of Louisiiiua's 
Vocational and Technical Education l^ogram- Fiscal Year 1972 (Baton Rouge: Louisiana 
State Advisory Council for Vocational and Technical Education, 1972). 



Putpiitt III tilc Uv^hMtifin . In addition to lullillinj> (Itc icquirtmcnt^ of tlit Vot;itional 
r.JiK^tinti Aincn<)nu nt^ of 1 the ev,iluatiMn huA tlirtr fiddumnal piirp'ncs: 1 11 prnvulc 
itiioriiMtKn on Ijiui^uiu^ Votsticnul a\u\ Tct linital Udutation Projtiim, including it^ ^trcngth^ 
A\n\ pij^^iMc wcaknc^H^^ from whitli the St.itc Advisory (Council c;in draw tontlu?;u)iH and 
make rccomiiicndatioiu: (2) to ^t\c a hasi«i for reporting to the citizen** of Louisiana by 
the State Advuory ('ouncil; and (3) an cvahntion of the degree to which Louisiana has met 
the neet!^ of eiiiph^yers and individuaU for votational and tc^. ^nical education" (Strong, 

Mc!ho(ioh)t^v . n^ta for thi«. Myuiy were cojnpiicd from j^tate report*, other available 
studies (Muluding state and national i^tudie^), and from discvi«»sit)ns and interviews mainly 
with MAic V(icati(jn;il education staff membcts. 

Results. The survey staff made 1 3 rec.Miimeirdations upon completion of their study. 
Some of these retoninicndations are possibly of interest only to the state. Those recommen- 
dations likely to be of general interest nicludc: 

(1) *' rhat the l.ouisi.»na State H«)ard of Education and Stale Superintendent be encour- 
agC(l to devclt>p a long rangf master plan for vocational and technical education'* svhich wt/uld 
require taking into ci)nsi(!eration total manpower training needs and a related plati for fiscal 
expenditures^ 

(2) *That guidance and counseling be strengthened at all levels of education, K througli 
adult" so that students may be better prepared to make future decisions. 

(3) "That every effi)rt be made to keep vocational cducatif>n in Louisiana ^brea^l of 
labor market trends/* In spite of a lack of precise data related to labor needs, programs must 
be justified based upon projected manpower needs. 

(4) *'That evaluation prt)ccdures be implemented to monitor Exemplary Programs so 
that the most svorthsvhile attribvites of the prt)jccts may be disscmirated for the general use 
of all sch(K)ls/' 

(5) "That greater communication and articulatior* occur between secondary schools, 
post-sect)ndary vocacionabtechnical schools, and colleges and universities in regards to curric- 
ulum and the needs of students" to assist students transferring from one instituiitm tti antither. 

(6) "That more extensive follow-up studies be instigated on both the secondary levels" 
and not be limited to only three months after graduation but to be extended to at least on? 
year after graduation (Strong, 1972), 



Trie vitinn Pr<»e^^nn 



M^iMv timc^pf'v^t^iti. iUsihypa] tn ih^ngr ^ttitudc^. TIh^ h example nfhow one 
ilutt.t^ dctcfiiniicH ^ttitiulin^l Jiriiigi ^3 

ProMt < hrrvfiH. In ^n tllntt tncntonr^K^ ninie MiKlrnt^^ tn cnmll in vot^tnnul 
tcclinual rJiK^tinn onif^f.^. tlie M^/iltnn, Vaw^yU'Auu. Si Unn] DHifut tcUc«j tlic effective. 
uc%% ifi l^pca tcKvi^nni pfMgt^tin ni ni(. t„ung Mndcnt^^ jb^^ut y h^ ;iiul < ^fcrt^. The lnpot),cM% 
w^MlMt ptuitnc attituJc* tnvv-.f,l ^i UaiA L?Jiif^tnu]i\ In HcvdMpnl ni diiMrm hy having 
Jlu ni view tA^i A ttli vi^inn prn^f^nu ^prMfually iliMgntcl tn infofm them nf vaf nnj% a^pcc 
of cMili i ^ttrf. Tlu' prnjiit %\,4wniicH4t »rnpf'*vMt^: ilu' ^»w^f i nc arrrf ^wafcnc<»w>f 
«.i^rli ^r^ilc UwJtnt^ ^ml tli- j.ih nf unt^n^n nf nnitli gf^*)c %tuMriitv Tut jnli ^rca*, were 
vclcitcJ: il j lUt^ PfMiit.Mn^, (2) l^f^lhn^, {}] ImfA Scfvuc, (4) Pfintini^/Cw^i liir Art*., ^ml 

I'nrpuM- of ilif l.v^liMtnwi , Thr cvaliMtt Mi w^% Jcugn* to (U trrnmu wlut ^ttitudnu) 
ch^npc% nt<urr(1 ni MnJcntv afttr vicwnig ^ t^pctl tclrvjMon puurnt4tic,n about a ^prcific 



own a 



Mctlioijcjlctty, A rantlt>tnly -rlr^tcH t.4inplc *»f So until gi^ilr Mudmt* wnc 
taped tclrvuKHi ptcvcntatioii dvMrtnrd impart infMrmafH>n 4l>mjf « j(,l> ^ira. A film w 
Oumn ft>r cat li .if thr five Jtcax, StuMcnt^ wrrr pict( Mrtl imnirdutrly bcfc^ir and p.At 
\c%\ci\ itntiiciliatrly after ratli pirvr nt^lion. Tlu tmtriinicnt ^ka% tlir mmic for Iv^tli ptrtrvt 
atnJ pt>%ttCM 4tu\ totUKtcd i»f IH v to iaIiuIi flir Mudrnt^ wrir a%knl t<* r< t.pond umii^: a 
Ltkcrl type Tatli %tinjrtii\ u t»tc W4% taken a\ tlie mjiii of ilic iiulnujiul item re»p»»nM:^, 

A faiuioinly vrletteii ^niple o< SO 1 » mt li j-^tacle Minlentvwav MiinUrly tieatrd enttpt tliat 
%tuilent% wetc %lmwn pteventationv dc%ighc to ifiip^i t inf**inutic,n at ilie |ol> oiieiit^tKMi 
level ami they aUo partuipaied in a gtoup tiutuvMtni imniedutely after eatli pie^entation. 
Sc4(t*tual %ipfiifu4tHe WAS tJetettnitied iiy ^lihimtting r^Ji ^^ct ndiMA to a "t" teit an^lvui 
for Ciitf elated v^tMluev. "Ttie vaniple vaf Miue was uicd i»> Mati%tiv ally exauinu wlietlui of 
not there was %if;nifKani differeiue between the %ainplc prrtcvt attitudnial varul>i1ity ^fid the 
Mmple^)iuttc%t attitudinal vanabiliiy" (Uernahei. 1972;1.^). The null hypotheu^ ui r^Ji cave 
W4%: S-prete\t S-^piKttei.r ^ O, 



^Raytnoiul Uertuhei aiul Jaine^ C i .tfrtf Aw^iftnr^f^if Joh OfiroMfi.Hr vi.i i\ipc^J 
VWr-nVum /♦n^v^.i/ti, Uxperimental Study (t >i^yle-tcwn, IV\: Uu^kx (\igntv Puhlu SJmoU, 
1972). 
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He^uhcj. At the MXtli ^t^i\v level. '* . . . \h€ null liypotlucii^ w^ci rcjctted at (lie .025 level 
for (he VtuMiUpiitaphn Art*; auA WcKling jnh ^fea<i. M (he .25 level for (he Data Processing 
ji)h afc;*. iiiiil thctc iiuuf<uient eviiU lue to tcjcct the mill liypoth< ai* for the Hood Service 

Drafting jf»h area^" .Berruhei, 1972: 13). Loo^icly in(crprc(c(K thi!^ n?c;ins thai (he taped 
(clevi«.ion jne^cntatt«>nci tlul ;»ffett ^(iiilcnt a((it\Mlc<i only in (he Prin(ing/ni ;»phic Arts, Welding 
and l^AiA Proces^in^ afeas. 

At (he nin(h pnAc kvel, "(he null li yp(i( he^s was rejecd *l ri( the .025 level for (he Food 
.Service ji>b area. a( (he .05 level for (he Printing/Cir^iphic Arts job area, at the .25 level for (he 
Drafting j^>b arc^, and there wat nr-ufftticnt evidence to reject the null hypotheMii lor the 
Data Processing and Welding job areas** (Uernabei. 1972). Again* loosely interpreted, this 
means ihat sfudcnt attitudes were affected cucept in the areas of Data Processing and Welding. 

CVAtI Acadeniu C^uru nhnn Pr^irtt 

This study IS an example of an evahiation to determine tlic extent to which a prograrn 
nut Its specificillv slated objectives.'^ 

Prtnect Ovrrvieu . The •»cc»jn(l year of the Ovirdinatcd Vocational-Academic Education 
(CVAI;) Acatlemic Curriculum Projett was i)rv<>(C(J to the deveh)pnient ancJ field testing of 
individualued turr cula at the junior high level, ('urricula tncludcil mathematics, social studies* 
jitid language art> at the eighth grade level anti mathematics and science at the ninth grade. 
The field test took plate in the Austin Junior High Sthool, Phatr-San Juan^Alamo Independent 
Schotd DH(f4v f . m the lower Rtodrand Valley of Texas. 

P uu>osr lii the luuhuiiun . The evaluation deterninied the extent to which the project 
was nu'cting its goaU. 

Mcfh<»dMli>^»y . The Acid lest was tonduclcd with \ix teachers and approximately 120 
siudrnu wbo wetc ptcdoininantlv Mesu aii Amenc-tn and economically disadvantaged. At 
(he beginning: am! end of the field test psniod (the first semester of the 1971-72 school year), 
students i oinplcted a short v|uesiiunnaif e deM|»ned to measure their: (1) preferences for 
nhool subjects, [2) self concept of scht>ol ahtltiy, and (3) altitude toward school. Periodic 



"^l^dgiatii^n Setvue Center Kcgi»>n 1, ('.WMU^oofilinitttJ I'oc^ttioiuthAciKicmic luhca- 
nofM/M<M</f-miV C unicult$m hi\ifcl li\'*tlui\ticut H^^poff, 1971 72, (Edinburg, TX: Educativ)n 
Scrvue (tenter Hcjjum 1» 



classroom obscrMtioiis were iiiiule to (Icltrininc the degree of iinplcnicntation of the curricula 
and die type and extent of student ;ictivities. D.ita >ucli as school attendance and student 
discipline were obtained from school records. Curriculum imbedded criterion-referenced tests 
were used as a measure of student achievement. 

Results. Studentr' attitudes toward school and self-concept of school ability showed 
no statistically significant ^ains. A significant number of ninth gr;ide students changed their 
opinion of science from **dislike*' to **like." Classroom observation data showed that the 
curriculum was individualized and scudefits spent a large portion of their time in individual 
activities. The rate of abscMteeisin dropped from 8.9% to 7.9% and the rate of CVAE disci- 
plinary referrals dropped /rom 28.4% to 1 1.7%. **Students demonstrated martery of specific 
performance objectives, 'rom 88.7% to 99.8% in the different curriculum areas. More signif- 
icantly, they demonstrated short-term retention rates which varied from 33.2% to 77.4%" 
(Education Service Center, 1972). 

Evaluation Model for Career Programs 

After developing an innovative curriculum, a district is often left with the problem of 
how to validate the appropriateness of the curriculum. Here is how one local agency handled 
that challenge. 5 

Project Overview . This project was designed to develop ;nid test a model for qualitatively 
evaluating career programs. The results of the evaluation specify curricular areas possibly in 
need of revision. 

Purpose of the Evahi;ition . To determine the effectiveness of the evaluation mode! under 
actual conditions. 

Methodology . The two career training areas chosen for this study were (1) dental assis- 
tant, and (2) auto mechanics. The curriculum in each of these areas wus written in a forniat 
which included precise behavioral objectives. To determine the appropriateness of the cur- 
riculum in each training area, a population of i*itcresied and involved judges was identified. 
This population consisted of the following six groups; (1) students. (2) instructors, (3) 
advisory board, (4) graduates of the two training areas, (5) employers of graduates from the 



^Richard L. Bycrly, et ah, ^imluatiou Mo(hl for Career Proj^rdms, Final Report (Ankcny, 
lA: Des Moines Area Community College, 1972). 



7 



two training ^iic^s, and (6) busintssnicn who liavc not yet, but may in tlic future, hire graduatr's 
of the fvvo training areas. A population of the above mentioned judges was identified for 
each of the training areas. The judges were asked to rate each behavioral objective in their 
traiiiing area using the following scale: 

0 " of no importance 

1 - desirable but not necessary 

2 - desirable and necessary 

3 - absolutely essential 

Each of the training areas was divided into subareas. Group respondent scores were 
combined for each snbarea and a mean was computed, An analysis of variance was then con* 
ducted to determine if any significant differences existed, if sig.iificant differences were 
found, paired "t*' tests were used in an attempt to determine which groups were responding 
differently. 

Results. Several sub sections in each of the training areas weic found ro contain signif- 
icant differences. For example, in the electrical subarea of auto imchanics, tlic analysis of 
variance was significant at the .05 level. The paired **t** tens indicated that the employers, 
students and ex-students were placing less emphasis on these items than the other three groups. 

Using this approach, there were several .;reas where significant differences did 
exist between groups. These differences illustrate that the six groups perceive 
the tasks at different stratas of importance. The implication, tTuTelore, is that 
curricular revision may be necessary in these areas where divergence was apparent. 
If revision is not necessary, at least closer examination of the curricular differ- 
ences could be accomplished (Byerly, eC al., 1972:40). 

Project VIGOR 

This study made extensive use of the written cjuestionnrnrc to obtain information. It 
a\o has the rather unusual charactcristir of having a student conduct a portion of the eval- 



^^David Douglas Public Schools, Projvci i^lGOR: I'^ocatiofud Cluster /;\/ncafiOM, Inte- 
grated and Articulated Grades t ^I'hrough 14 with Guidance Services^ Occupational Ilxploration 
and IVork llxpcrience Relevant to General Uducation, Final Evaluation, (Porthmd, OR; David 
Douglas Public Schools, 1972), 
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Project Overview . Project gonls extend from career nwjircncss iit the primary grades to 
student follow-up ami placement at the post high school level. 

Through Project VIGOJl, the David Douglas Public School System (Portland, 
Oregon) is addressing itself to the objective of changing a conventional academ* 
ically-oricnted, general education school system into one whose curriculum 
reflects the needs of all students regardless of entry into their chosen vocation 
(David Douglas Public Schools, 1972). 

Purpose of the Evaluation . 

The evaluation was purposely designed as a very subjective, human oriented 
evaluatior strategy. The overall purpose was to gather as much information 
as possible about current teacher and administrator knowledge, feelings, and 
attitudes about project status for developing base line information for more 
sophisticated future evaluation (David Douglas Public Schools, 1972; 9). 

Methodology . Eight separate questionnaires were developed to ellicit responses relevant 
to project goals and/or objectives. These questionnaires were given to two different groups— 
administrators and teachcr-counsclors-at four different levels; elementary (grades 1-6), 
middle school (grades 7-8), junior high (grades 9-10), and senior high (grades 11-12). The 
response ratio was 60% from teacher-counselors and 85% from administrators. To verify 
responses obtained from the questionnaires, an interview was conducted with 33 David Douglas 
teachers, counselors and administrators. In addition to the questionnaires and interview, 
student information was collected by a member of the senior class at David Douglas High 
School. The student information was relevant to the knowledge and attitude of ninth grade 
students toward career education. 

Results. The findings and conclusions reported were substantial; those of general interest 
include the following. Awareness programs at grades 1-6 may become the unifying thrust 
needed for other components. However, "two apparent inhibitors of program changes within 
the institution: (1) the contention that this isn't really new, just a different language; and 
(2) the conviction that *this too shall pass away if we all just ignore it' " muse be reconciled 
(David Douglas Public Schools, 1972). 

The evaluator recommended that these inhibitions **might be overcome by evidence of 
positive results from the career education program" (1972). It was interesting that most 
administrators appeared to understand the thrust of the project while few teachers did. While 
involvement of employers and community groups was successful, there were limitations on 
the range of experiences which could be provided students due to the physical makeup of 
the local community. Considerable more effort is needed to increase ninth grade students* 
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knowledge of the career education concept and its relation to subject areas. To allow for a 
more comprehensive evaluation of the project, **indicators of performance and/or accomplish- 
ment need to be detailed for each project component" (David Douglas Public Schools, 1972). 

Central Kentucky Vocational Education 
Evaluation Project 

This study had several purposes including the development of a formal model for eval- 
uation of vocational education.^ 

Project Overview . The Central Kentucky Vocational Education Evaluation Project 
(VEEP) was a pilot project designed to develop expertise at the appropriate level for organiz- 
ing and implementing an effective evaluation of vocational programs Its objectives v;ere to: 

1) identify new or improved procedures for assisting schools in conducting program 
evaluation, 

2) test and demonstrate evaluation procedures to determine whether the secondary 
and post-secondary programs of vocational education in Central Kentucky are 
fulfilling the stated objectives, 

3) develop state and local leadership competencies needed for evaluating programs 
of vocational education, and 

4) enable personnel in the local and regional schools to be more proficient in the 
various tasks necessary for an adequate evaluation. 

The project focused on product evaluation which emphasized students' attainment of criterion 
behavior as stated in objectives. The project activities consisted of conducting conferences, 
maintaining consultive services, developing an evaluation manual, analyzing school data, and 
preparing project reports. 

Purpose of the Evaluation . The project was evaluated t ^ determine the extent to which 
it met its stated objectives. 



^Kentucky Research Coordinating Unit, Central Kentucky Vocational Education Eval- 
uation Project, Final Report (Lexington: Kentucky Research Coordinating Unit, October, 
1972). 
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MethodologY- Seminars conducted by the VEEP staff were evaluated by participants 
responding to questionnaires, VEEP staff members made periodic visitations to participating 
local school districts to observe the evaluation efforts and to interview local leadership teams. 
Each participating school district was asked to prepare quarterly reports of their activities 
which supplied further evidence of local involvement. At the conclusion of the project, a 
follow-up questionnaire was sent to ail participants to determine their perceptions of project 
effectiveness. 

Results, A formal model for evaluation of vocational education was adapted from several 
sources including those developed in Michigan (Byram, 1965), An instrument to obtain follow- 
up information from former vocational education students was developed and utilized by 12 
participating school districts. Several workshops were conducted for school district staff 
members to assist them in acquiring the abilities and knowledges necessary to perform an 
adequate program evaluation and develop measurable performance objectives. The project 
also developed and maintained effective public relations through the use of various media, 
including a newsletter, 

SOME CONSIDERATIONS FOR 
PROGRAM EVALUATION 

For discussion purposes, the planning and evaluation system used in the VEEP project 
(Kentucky Research Coordinating Unit, 1972) will be used as a guide when discussing some 
of the things one should consider when evaluating a vocational-technical education program 
(see Figure 1), 

The reader should understand that this is only one example of a system or model for 
evaluation. Many different systems exist, varying in complexity, design and intent. For 
examples of other systems, the reader is referred to Provus (1969), Stufflebeam (1971), EPIC 
Evaluation Center (n,d,), and Stake (1967), This is not an exhaustive listing of evaluation 
systems, but it does provide for a wide variance in methodological approaches. 

There are at least two important considerations when choosing a system for evaluation: 

1) Is the system appropriate for the situation to be evaluated? 

2) Are the evaluator and others concerned capable of handling the complexities of 
the system? 
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FIGURE 1. -PLANNING AN EVALUATION SYSTEM 



It is uncomfortable and counter-productive to be saddled with an evaluation system too com- 
plex for some of the participants. As Majer (1972) points out when referring to a situation 
involving the CIPP model: 

One of the basic, pervading problems was the use of a moderately complex model 
for evaluation, the CIPP model. This is no indictment of the model. It is based on 
sound theory and can be very useful in program evaluation. The problem rather 
was a result of the model being somewhat complex to use with participants (staff, 
graduate students, and school personnel) whose professional expertise is in other 
areas. It is not a model for people untrained in evaluation. One bit of evidence 
for this was the continued confused discussion (and resulting frustration) about 
whether a given activity was "context" or "input" evaluation. In reality this is not 
an important question. The important question is whether one can develop a strat- 
egy for getting valid information necessary to make decisions . . . . " 

Majer suggests choosing a rather straightforward system for evaluation and examining some 
of the considerations along the way through the system. 

Assess Needs 

Kaufman contends that, 

the identification of needs is a discrepancy analysis that identifies the two polar 
positions of: 

Where are we now? 
Where are we to be? 

; and thus specifies the measurable discrepancy (or distance) between these two 
poles (1972:29). 

The actual assessment of needs can be accomplished in any number of ways from a 
community analysis (Cromer, 1968), to a determination by authorities. Regardless of how 
needs are assessed, it is important to the planning process that the information gained is valid. 
From the discrepancy identified through the needs assessment, an idea is formulated to, hope- 
fully, remove the discrepancy. 

Develop Philosophy 

Before a remedial program (to overcome the identified discrepancy) can be developed, a 
basic guiding philosophy should be developed. One of the concerns for such a philosophy 
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should be: **Wlictlier or not the remediation can be effected entirely through the effort of 
the local agency or should outside assistance be recruited?" Perhaps the decision is made to 
adapt some innovation developed elsewhere to fit the local situation. In such a case, an 
evaluation scheme such as that reported by Hull and Wells (1972) would be appropriate to 
assist in choosing the correct innovation. Whatever philosophy is chosen for guiding the 
development of the remediation, it should not be viewed as inviolate, but changeable if evi- 
dence is presented indicating a need for change. 

Write Objectives 

Now that a need has been determined and a guiding philosophy developed, program 
objectives should be identified. These may be written, as in the case of a locally developed 
program, or adapted from an existing program. Regardless of the source, as Bloom (1971) 
points out, to facilitate the evaluation process, objectives must be **meaningful, unambiguous 
statements of intended educational outcomes." Often objectives are written that only identify 
inputs rather than outcomes. An example might be: 

**Staff development will be provided for teachers." 

or 

*The students will view a 30 minute film." 

This type of an objective can be evaluated quite easily by verifying the input, but unless some 
outcome is specified, the evaluation cannot say whether or not any desired results occurred. 
The program objectives will determine to a large extent the evaluation that is conducted; 
therefore, time and energy spent in their careful construction will bring many benefits. 

If the program is of an instructional nature, then the research reported to date suggests 
that behavioral objectives are probably beneficial (Walbesser, 1972). There seems to be a 
positive effect from having the learner informed of the expected learning outcomes, and 
learner knowledge of the objectives seems to increase the rate of acquisition as well as decrease 
the rate of forgetting. A number of publications are available to assist in developing behavioral 
objectives, including Popham (1970), McAshan (1970) and Walbesser (1970). Critics of 
behavioral objectives (Eisner, 1972) generally see them as producing too narrow a view of 
education. 

State Criterion Questions 

An evaluator has been characterized as an extension of the mind of the decision-maker 
(Stufflebeam, 1971). His role has two major components-the technical and the interface. 



14 



It is within this interface role that the evaluator must couch the evaluation's criterion ques- 
tions. Scriven (1967) has typified evaluation as attempting to answer certain types of ques- 
tions about various educational processes, programs, and the like. 

In the evaluation system used in this paper, the evaluator, within his interface role and 
by paying attention to the program objectives, can develop a meaningful list of criterion 
questions, Welty ( 1 970) describes the painstaking steps necessary in the development of a 
series of questions to formulate an evaluation design. He advocates that the evaluator meet 
personally with people interested in the program so that they will have input into the develop- 
ment of the evaluation design. 

It is the opinion of this author that a formal document should be made of the criterion 
questions, along with all appropriate information relative to what data will bo collected for 
each question, by whom it will be collected, and when it will be collected. This document 
should then be approved by the appropriate authority, i,e., in the case of a local school 
district, the school board. For an example of an evaluation des^n, the reader is referred to 
Fielstra (1971), 



Collect Data 

The collection of data and its subsequent treatment can be classified generally into two 
major activities. The first is the monitoring of the project as it is implemented. The second 
is measurement of attainment before, after and as many times as necessary during the project. 
These two activities have been variously defined and named in the literature. Scriven (1967) 
and Bloom (1971) refer to them as formative and summative evaluation, Stufflebeam (1971) 
and others refer to them as process and product evaluation. For the remainder of this paper, 
the latter classification scheme will be used. Process evaluation is explained as providing 

. , , periodic feedback to persons responsible for implementing plans and proce- 
dures. It has three objectives: (1) to detect or predict defects in the procedural 
design or its implementation during the implementation stages (2) to provide 
information for programmed decisions and (3) to maintain a record of procedure 
as it occurs (1971:229). 

Product evaluation "measures and interprets attainments at the end of the project cycle and 
as often as necessary during the project term" (Stufflebeam, 1971:232). 

Certainly both process and product evaluation are needed. As Glaser (1972) said in 
discussing his model which he calls the "clinical approach": 
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assessments of research or service efforts will be most meaningful and useful 
if they are designed to evaluate the program as a whole and as a dynamic, 
complexly interrelated entity. 

If a program should be assessed by product evaluation only, a great deal of risk exists in not 
knowing what the program really was, 

Norton was certainly correct when he said, "to evaluate process is much easier said than 
done" (1972:62). However, authors such as Eisner (1972) make a strong case for the careful 
observation and description of the program environment. It has been the observation of this 
author that careful attention to process evaluation has at least two major redeeming qualities 
which make it worth the large expenditure of resources. First, it demands that the evaluator 
become well informed with the functioning of the program and the complex interrelatedness 
of the many variables involved. Second, information provided from periodic observations 
serves to alert management to any design-implementation discrepancies. 

Product evaluation needs little if any justification, since it is aimed at the measurement 
of program attainments. However, a thorough consideration for the many data sources for 
product evaluation is necessary for a comprehensive evaluation (Webster, 1973). 

There are many types of data that can be collected and different methods of collecting 
them. Some of the more common ones are discussed below with additional sources of infor- 
mation included where appropriate. 

One of the more obvious methods of data collection is researching the school files and 
records. These files include such items as student cumulative records, and personnel, business, 
and attendance records. As with other data, these data should be handled in such a way so 
as to insure individual anonymity. Some of the types of data which can be obtained by 
searching existing records are: 

1) student demographic data including birthdate, grade point average, attendance, 
address, and test scores; 

2) teacher demographic data including years of experience, major and minor field, 
and degrees held; and 

3) school demographic data including enrollment by grade, by section, average 
daily attendance, and schedule of classes (Webster, 1973). 
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Individual and group testing is one of the most widely used methods for obtaining eval- 
uative data. There are a number of sources available to the reader which discuss thp various 
tests that are presently available. Probably the most well known is the Mental Measurements 
Yearbook series (Buros, 1972)* 

Findley (1971) has an important discussion of factors that affect test results. He points 
out the importance of test administration, scoring and recording as they effect the results. 
It is important that the evaluator monitor at least a sample of each test administration to 
identify any possible problems. Clemans (1971) presents a list of general recommendations 
for proper planning of test administration which shopld assure smooth testing procedures. 

As Seibel (1968) points out, achievement tests, scholastic aptitude tests and intelligence 
tests are quite similar. 

In fact, if we were to look at an item from a test without knowing its source, 
it would be difficult and perhaps impossible to specify with assurance whether 
the item came from a test that is labeled as an achievement test, a scholastic 
aptitude test, or an intelligence test (1968:265). 

Of the three groups of tests, perhaps the type of test most often used for program evaluation 
is the achievement test (Downie, 1967). 

A second group of tests are those classified generally as attitude— interest tests. Partic- 
ularly the vocational interest tests have proven effective to counselors. Downie (1967) 
identifies the problems with attitude tests as: 

1) Low correlation between measured attitudes and observed behaviors; 

2) Low consistency between specific items and general concepts; 

3) Change in attitudes over time; and 

4) The interpretation of neutral attitude. 

He also identifies some problems with vocational interest inventories as: 

1) They can be faked. 

2) They have a high level of reading difficulty. 

3) The possibility exiles of always receiving socially acceptable responses (Downie, 
1967). 



17 



A third classification of tests arc criterion-referenced tests. Brazziel (1972) explains 
criterion-referenced tests as measures of **studeiit progress toward explicit objectives as 
defined by the school enterprise. They are measures of degree of mastery of material taught 
and learned in a specific time frame.'' Thus, from a student's incorrect response to a criterion- 
refenMiced item or set of items, a teacher can prescribe further remediation for the student by 
referring back to the antecedent objective. A number of test publishers arc currently publish- 
ing and marketing criterion-referenced tests. 

Questionnaires and interviews are also widely used methods of collecting data for eval- 
uation purposes. Phillips lists the relative ease with which an evaluator can obtain a great 
deal of information from a probability sample as one of the virtues of tliese methods (Phillips, 
1966). Questionnaires and interview schedules can vary from open-ended to closed-form 
instruments. The closed-form instrument is advised when categorized data is nee'led, whereas 
the open-ended instrument is best suited for preliminary exploration of untried situations. 

The primary considerations in developing questionnaires and interview schedules are: 
(1) keep the questions simple and concrete, (2) questions should not be open to subjective 
evaluation, and (3) do not bias the response (Young, 1966). Young contends that a carefully 
worded questionnaire provides privacy and anonymity while an interview conducted by a 
well trained interviewer is a highly flexible tool. Either method, of course, should be pilot 
tested before using. 

The author has found that interviewing a sample of respondents and, if possible, non- 
respondents to a questionnaire survey is well worth the extra effort. Verification of how 
subjects are interpreting questions can be obtained by interviewing respondents and an indica- 
tion of bias introduced from not having a 100 percent return can be determined by inter- 
viewing non-respondents. 

Direct observations are another method of data collection. Although this method is very 
time consuming and thus costly, the author has found the information gained well worth it. 

Roberson (1970) presents six different observational categories. This discussion will be 
limited to on-site activity observations or **what people do." He gives several recommenda- 
tions for selecting or developing an observation system. 

1) The observation system should include terms. 

2) The terms should be simply defined. 

3) Examples of the activities to be observed should be given. 
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4) Tiic observer slioulci be provided an easy furiiiat for coding observations (1970). 

Phillins (1966) rccommciuls combining the interview technique with observation. The 
two methods complement each other since interview data provide what the subject says he 
docs while observation data provide information about what he actually dues. 

From the experience of the author, it has been found that when observing chissrooiDs, 
teachers appreciate knowing what the observer is doing. Results of observations can be dis« 
cussed with teachers; and if observations arc to be maintained over a period of time, better 
rapport will result. It :3 realized that there is a danger of biasing the situation by telling the 
teacher v'hat is being observed, but experience has shown that when classroom activities are 
being periodically observed, teachers tend to ignore the observer. 

Another method of data collection is through the use of unobtrusive measures (Webb, 
1966). This type of measurement is difficult to design. An example is measuring the extent 
that books are read by judging the amount of wear and tear on the book itself. 

Regardless of the method usci for collecting evaluative data, there are different means 
of collecting it. Teachers can collect it (Education Service Center, Region 1, 1972); students 
can collect it, and internal evaluators can collect it (David Douglas Public Schools, 1972); 
and the community can provide it (Byerly, 1972). 

Two separate types of vocational education program evaluation require a little different 
collection of data or possibly a different use of data. Follow-up studies are designed to deter- 
mine what former students are doing after leaving the program and how the program effec- 
tiveness is perceived. Voelkner (1971) lists the variables of interest in a follow-up s'-udv as; 

1) length of time a graduate takes to find a satisfactory job, 

2) employment security, 

3) length of time on the job, 

4) earnings progression, and 

5) rate of advancement. 

Wilson (1971) sees the follow-up of students as a way of increasing vocational education 
accountability with the community by making business and industry more aware of the pro- 
grams. Follow-up studies also provide information relative to program changes (Jacobson, 
1971;Elson, 1972). 
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C'ost cOcctivciicss studies arc an attempt to attach costs to progniiii inputs and/or out- 
puts. Kaufman ( 1 provides an introductory discussion of the subject. Connor { 1 972) 
purports to sliows how cost-henefit analysis can be used to determine the most effective way 
of combining Vocational education with general education. Stanton (1970) shows how the 
costs of vocational education arc far less than for manpower programs. It is the opinion of 
the present author tliat trying to atiacli costs to program inputs can be accomplished with 
reasonable satisfaction. However, tryitig to attach costs to program outputs is very difficult 
because of problems of definition and measurement. 



Analyze Data 

There are a host of books related to methods of statistical data analysis. Two excellent 
general texts are by Hayes (1972) and Glass (1972). There arc also a number of books that 
discuss the use of computers in analyzing statistical data. An introductory one is by Vcldman 
(1967) while Coolcy (1971) has a more advanced one. The author will not attempt to discuss 
the many ways available to analyze data; however the cvaluator should ascertain that the 
analysis is appropiatc to the data collected. As a matter of routine, all computer programs 
should be tested on a set of sample data before being used on actual data. 

Since the present system for evaluation being used for this paper docs not indicate where 
reporting of data should appear, the author has chosen to insert it at this point. It has been 
the experience of the author that unless the report is written in a brief, easy to understand 
manner it will turn more people off than on. If a sophisticated statistical analysis is needed, 
by all means than the cvaluator should do ir; but when the report is written, the analysis 
should not so confuse the reader that the results arc lo**t. An **cxccutivc summary" similar 
in form to that used in the second section of this paper should precede any report, This will 
facilitate understanding on the part of the reader. 



Form u I a t e Recommendations 

The evaluator is obligated to provide any information suggested by the data as possible 
niethods for improving the program (Cameron. 1971), This author concurs; however, the 
cvaluator must be cautious and recommend only that which is suggested by the data. The 
recommendations should appear in both the executive summary and in the text of the report, 
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Make Decisions 



Hopefully, through the use of process evaluation, information has been provided to 
management since the inception of the program and many decisions have been made 
(Stufflebeam, 1971), The evaluator should not be surprised if a r'^commendation is not 
acted upon; there is a political dimension to the decision-making process. 

Decisions can be made to start the evaluation cycle all over again by reassessing needs 
or go to some other phase of the cycle. 

SOME FURTHER CONSIDERATIONS 
ABOUT PROGRAM EVALUATION 

There are two aspects of the evaluation of the Skyline Career Development Center in 
Dallas, Texas that seem to extend beyond anything currently found in the literature. The 
Career Development Center is a larfj;e complex housing 28 career clusters. Any senior high 
student in the Dallas Independent School District is eligible to apply for enrollment in any 
of the career areas. Transportation from each of the local high schools to the Career Develop 
ment Center is provided. The cluscers are organized in a three hour instructional block of 
time so that a student is only in a given cluster for either the morning or afternoon session. 
The instructional staff consists of 103 instructors for a student population of about 2,350. 

Context Evaluation 

In an attempt to get an accurate profile of the Career Development Center student pop- 
ulation, a great deal of information on each student is obtained and recorded in computer C\U 
The following is a list of all data sought for each student. 



1) 


Identification number 


2) 


Home school 


3) 


Birth date 


4) 


Parent name 


5) 


Address 
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6) 


Phouc number 


7) 


Sex 


8) 


Ethnic background 


y) 


Grade level 


10) 


Cluster— (course) 


11) 


Period of day (A.M. or P.M.) 


12) 


Parent occupation 


13) 


Mode of transportation to Skyline 


14) 


Scores from 9th grade Iowa Tests of Educational Development 


15) 


Scores from 8th grade California Test of Mental Maturity 


16) 


Social Security Number 


17) 


Previous Career Development Center attendance 


18) 


Scores from Armed Services Vocational Aptitude Battery 



T^e information within the data base is updated as changes occur. Computer generated 
analyses of the student data are produced in the fall, at the end of the first semester, and 
again at the end of the school year. With the computerized data base it is relatively easy to 
generate a random sample of students, report periodically on the withdrawal rate, and provide 
instructors and administors with different types of information needed for annual reports. 

Upon graduating from the Career Development Center, a student's record is moved to 
the graduate file which is used to conduct an extensive follow-up evaluation. Currently this 
information is being used to research cost-effectiveness as it relates to obtaining follow-up 
information. 
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Process Evaluation 



In order to provide information relative to program implementation, classroom activities 
arc regularly monitored by trained observers. Three rounds of observations are scheduled 
each year, A round of observation is defined as the length of time necessary to observe all 
clusters. For each round of observations a different set of factors are observed. The design 
for process evaluation is graphically represented in Figure 2, 



Hours 




Sep Oct Nov Dec Jan Feb Mar Apr 
DURATION OF PROJECT 



FIGURE 2. -NUMBER OF OBSERVATION vs LENGTH 
OF OBSERVATION PER ROUND OF OBSERVATION 



/^t the beginning of the project year, four observations of approximately 20 minutes 
each are made in each cluster. For this first round of observation, the factors being observed 
include such things as adequacy of facilities and materials. During this time, one short inter- 
view with each instructor is conducted. By the end of the year, the observations consist of 
one observation of about three hours duration for each cluster. During this time, students 
are interviewed to determine their attitude and perceptions. The reader must realize that 
Figure 2 is only a **road map" and in actual practice the number and length of observations 
will vary to fit the situation. The overall design seems to work well since the frequent short 
visits initially help the evaluator develop rapport with the situation and the various instructors. 
By the end of the year, the student information gained from the interviews is very valuable. 
Reports on process evaluation information are provided to management about every two 
weeks or sooner, depending on the frequency of visits. 
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EPILOGUE 



Kilidschy (1971) makes a strong point for the evaluation of the vocational education 
department by advisory councils. The author sees a great deal of value in using the business 
and industrial community as sources of information. What better judges can be chosen for 
program evaluation, particularly curriculum validation, than the final users of the product 
of vocational education, the employers. This type of validation, coupled with hard data to 
verify student and staff satisfaction, student achievement, and cost-effectiveness, will com- 
prise a comprehensive evaluation. 

Of course evaluation costs money, and the plea for funds has gone out (National Advi- 
sory Council on Vocational Education, 1971). If a justification for further expenditures is 
needed, then heed what Armstrong (1969:16) says: 

A primary defense against an undesirable reaction to the many good programs 
in Vocational Education is careful evaluation of results to insure the elimination 
of defective ones before they create a public scandal. 
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FORiWORD 



Vocational and technical education has enjoyed high visibility during the past few years 
and with it increased pressure to account for expenditures and to justify programs. As a 
result, educators are ever alert for effective means of evaluating their educational programs. 
This publication and its three companion documents {Personnel Evaluation in Vocational 
and Technical Education, Facilities Evaluation in Vocational and Technical Education, and 
Student Evaluation in Vocational and Technical Education) provide educational practitioners 
with a review and synthesis of the most important works in evaluation as it applies to voca- 
tional and technical education. 

In Program Evaluation in Vocational and Technical Education, the author previews 
selected studies which provide a cross-section of methodology used in evaluating vocational 
and technical education programs. The various phases of program evaluation are documented 
and context evaluation and process evaluation are addressed in terms of a real situation at the 
Skyline Career Development Center in Dallas. 

The profession is indebted to William T. Denton for his scholarship in the preparation 
of this report. Recognition is also due Gordon Law, Department of Urban Education, Rutgers- 
the State University; and Donald L. Rathbun, Associate Director, American Vocational Asso- 
ciation for their critical review of the manuscript prior to final revision and publication. Paul 
E. Schroeder coordinated the publication's development, and Alice J. Brown and. Paula Kurth 
provided the technical editing. 

Robert E. Taylor 
Director 

The Center for Vocational and 
Technical Education 

ERIC Clearinghouse on Vocational 
and Technical Education 
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INTRODUCTION 



The purpose of this paper is to present information relative to the state of the art in 
evaluation of vocational and technical education programs. The first section of this paper is 
not intended to be an exhaustive review of the literature; however, it does present some reviews 
of studies in the field of vocational and technical education that are related to the topic: 
evaluation of programs. These studies were chosen for review so the reader might obtain a 
feeling for the wide range ot methods by which programs are evaluated. The results section 
of the reviews demonstrates the diversity of information that can be provided through evalua- 
tion. 

The second section of the paper takes a formal model developed for the evaluation of 
vocational education (Centra! Kentucky Research Coordinating Unit, 1972) and explains the 
types ot information and methods which can be used to obtain the information at each phase 
of the model. Appropriate references are made to the literature to direct the reader to addi- 
tional sources of information. 

The third section of the paper is devoted to some suggestions which the author has found 
to be practical from his experience while coordinating the evaluation of the Skyline Career 
Development Center in Dallas^ Texas. 

The anticipated audience for the paper is the practicing educator who finds himself, 
because of either external demands or personal desires, needing to evaluate a program in the 
area of vocational and technical education. The paper should also prove useful to the admin- 
istrator who wants to learn more about the complex field of evaluation. 

It is the position of the author that, 'although the demand for program evaluation is often 
from an external funding source, the primary beneficiary of an evaluation is the local agency. 
. That is, the information provided by a comprehensive evaluation for the purpose of structur- 
ing a program at the local level should exceed any requirements of an external agency. This 
bias undoubtedly is reflected in this paper. 

For the sake of clarification, the definition of evaluation as used in this paper is that of 
Cuba (1968:23) "Evaluation is the process of . . . obtaining and . . . providing . . . useful . . . 
information for making . . . educational decisions.** 



ERIC 



A R12V1EW OF SELECTED STUDIES 



The following reviews of projects are chosen to exemplify the diversity of ways in which 
evaluation information can lie obtained. The methods of evaluation vary from statistical 
analysis to a review of documents. The personnel involved in collecting the information also 
vary from outside consultants to students. The important point in each case is that informa- 
tion was provided to effect the decision-making process. 

Educational agencies arc always on the alert for new, innovative programs and the local 
evaluator is often asked to provide inputs about the relative merits of innovative programs 
prior to pilot testing. 



The Classification and Evaluation of Innovatidns in 
Vocational and Technical Education 

The following study was designed to develop a model for evaluating innovations.^ 

Project Overview . The project was set up to develop and field test a taxonomy of inno- 
vations appropriate for vocational and technical education. A consequent purpose was to 
develop guidelines for evaluating innovation characteristics. 

Purpose of the Evaluation . Once designed, the classification system was field tested to 
determine how project personnel at exemplary project sites would rate individual items. The 
input thus provided helped determine the final system. 

Methodology . The Innovations Evaluation Guide was initially developed after a review 
of the literature and interviews with six "innovative'* superintendents and their staff. The 
guide was then pilot tested in a junior high school for clarity and readability, followed by a 



^William L. Hull and Randall L. Wells, The Classification and Evaluation of Innovations 
in Vocational and Technical Education^ Final Report, Research and Development Series 
No. 71 (Columbus: The Center for Vocational and Technical Education, The Ohio State 
University, 1972). 



field test at four exemplary project sites which were chosen by a branch of the U,S. Office 
of Education. The field test consisted of staff members at the exemplary sites rating each 
of the items in the Innovations Evaluation Guide as to whom the item was important: teacher, 
administrator, or both. Each item was rated as to how essential it was. State supervisors and 
local project directors of exemplary programs attending a national conference also participated 
in the field test. The data thus collected in the field test were used to make final revisions 
of the Innovations Evaluation Guide. 

Results. ''A comprehensive taxonomy which classifies discrete vocational-technical 
innovations into mutually exclusive categories was not possible with our present level of 
knowledge and technology** (Hull and Wells, 1972:39). 

Probably of more importance, however, is that the Innovations Evaluation Guide can be 
used by those faced with the enormous problem of finding and choosing innovations to meet 
locally identified needs. It is interesting to note that in evaluating innovations, respondents 
chose as the most essential items quantity of staff costs, availability costs, availability of 
dollars, space (housing), sources of dollars, hardware, an^l complexity. Those items chosen 
as least essential were; rate of learning, entry and advancement in an occupation, new rela- 
tionships among groups, cyclical considerations, economic and social efficiencies, reliability, 
and divisibility (Hull, 1972), 

The Annual Vocational Report of Louisiana's Vocational 
and Technical Program-Fiscal Year 1972 

Oftentimes an evaluation must be done from existing documents, especially in the case 
of state agencies. This study is an example of such an evaluation,'^* 

Project Overview . Any state desiring to receive funds under the Vocational Education 
Amendments of 1968, Public Law 90-576, must establish a state advisory council. Among 
other duties, the state advisory council must . . . '^evaluate vocational education programs, 
services and activities under this title, and publish and distribute the results thereof; . . 
(Strong, 1972:1). This report fulfills that requirement. 



^Merle E, Strong and Daniel Jarosik, The Annual Vocational Report of Louisiana's 
Vocational and Technical Education Program -Fiscal Year 1972 (Baton Rouge: Louisiana 
State Advisory Council for Vocational and Technical Education, 1972). 



ERLC 



3 



Purpose of the Evaluation . In addition to fulfilling the requirements of the Vocational 
Education Amendments of 1968, the evaluation had three additional purposes: (1) **to provide 
information on Louisiana's Vocational and Technical Education Program^ including its strengths 
and possible weaknesses from which the State Advisory Council can draw conclusions and 
make recommendations; (2) to serve as a basis for reporting to the citizens of Louisiana by 
the State Advisory Council; and (3) an evaluation of the degree to which Louisiana has met 
the needs of employers and individuals for vocational and technical education" (Strong, 
1972:3-4). 

Methodology > Data for this study were compiled from state reports, other available 
studies (including state and national studies)) and from discussions and interviews mainly 
with state vocational education staff members. 

Results, The survey staff made 13 recommendations upon completion of their study. 
Some of these recommendations are possibly of interest only to the state. Thoise recommen- 
dations likely to be of general interest include: 

(1) *That the Louisiana State Board of Education and State Superintendent be encour- 
aged to develop a long-range master plan for vocational and technical education** which would 
require taking into consideration total manpower training needs and a related plan for fiscal 
expenditures. 

(2) 'That guidance and counseling be strengthened at all levels of education, K through 
adult** so that students may be better prepared to make future decisions. 

(3) 'That every effort be made to keep vocational education in Louisiana abreast of 
labor market trends." In spite of a lack of precise data related to labor needs, programs must 
be justified based upon projected manpower needs, 

(4) "That evaluation procedures be implemented to monitor-Exemplary Programs so 
that the most worthwhile attributes of the projects may be disseminated for the general use 
of all schools.*' 

(5) 'That greater communication and articulation occur between secondary schools, 
post-secondary vocational-technical schools, and colleges and universities in regards to curric- 
ulum and the needs of students" to assist students transferring from one institution to another. 

(6) 'That more extensive follow-up studies be instigated on both the secondary levels** 
and not be limited to only three months after graduation but to be extended to at least one 
year after graduation (Strong, 1972)i 



Career Awareness/|ob Orientation via Taped 
Television Programs 



Many times programs are developed to change attitudes. This is an example of how one 
district determined attitudinal changes. 3 

Project Overview. In an effort to encourage more students to enroll in vocational- 
technical education courses, the Hazelton, Pennsylvania, School District tested the effective- 
ness of taped television programs in informing students about jobs and careers. The hypothesis 
was that positive attitudes toward selected careers could be developed in children by having 
them view taped television programs specifically designed to inform them of various aspects 
of each career. The project was aimed at improving the job awareness/career awareness of 
sixth grade students and the job orientation of ninth grade students. Five job areas were 
selected: (1) Data Processing, (2) Drafting, (3) Food Service, (4) Printing/Graphic Arts, and 
(5) Welding. ^ 

Purpose of the Evaluation . The evaluation was designed to determine what attitudinal 
changes occured in students after viewing a taped television presentation about a specific job 



area 



Methodology. A randomly selected sample of 50 sixth grade students were shown a 
taped television presentation designed to impart information about a job area. A film was 
shown for each of the five job areas. Students were pretested immediately before and post- 
tested immediately after each presentation. The instrument was the same for both pretest 
and posttest and consisted of 18 items to which the students were asked to respond using a 
Likert-type scale. Each student's score was taken as the sum of the individual item responses 
A randomly selected sample of 50 ninth grade students was similarly treated except that 
students were shown presentations designed to impart information at the job orientation 
level and they also participated in a group discussion immediately after each presentation. 
Statistical significance was determined by submitting each set of data to a "t" test analysis 
for correlated variances. "The sample variance was used to statistically examine whether or 
not there was significant difference between the sample pretest attitudinal variability and the 
sample posttest attitudinal variability" (Bernabei, 1972:12). The null hypothesis in each 
was: S^pretest - S^posttest = 0. 



case 



^Raymond Bernabei and James Case, Career Awareness/Job Orientation via Taped 
Television Program, Experimental Study (Doylestown, PA: Bucks County Public Schools, 
1972). 
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Results. At the sixth grade level, . . . the null hypothesis was rejected at the .025 level 
for the Printing/Graphic Arts and Welding job areas, at the .25 level for the Data Processing 
job area, and there was insufficient evidence to reject the null hypothesis for the Food Service 
and Draftingjob areas" (Bernabei, 1972:13). Loosely interpreted, this means that the taped 
television presentations did affect student attitudes only in the Printing/Graphic Arts, Welding 
and Data Processing areas. 

At the ninth grade level, *'the null hypothesis was rejecte u vhe .025 level for the Food 
Service job area, at the .05 level for the Printing/Graphic Arts job area, at the .25 level for the 
Draftingjob area, and there was insufficient evidence to reject the null hypothesis for the 
Data Processing and Welding job areas" (Bernabei, 1972). Again, loosely interpreted, this 
means that student attitudes were affected except in the areas of Data Processing and Welding. 

CVAR Academic Curriculum Project 

This study is an example of an evaluation to determine the extent to which a program 
met its specifically stated objectives*"* ^ 

Project Overview . The second year of the Coordinated Vocational-Academic Education 
(CVAE) Academic Curriculum Project was devoted to the development and field testing of 
individualized curricula at the junior high level. Curricula included mathematics, social stndies, 
and language arts at the eighth grade level and mathematics and science at the ninth grade. 
The field test took place in the Austin Junior High School, Pharr-San Juan-Alamo Independent 
School District, in the lower Rio Grand Valley of Texas* 

Pur pose of the Evaluation . The evaluation determined the extent to which tli^ ^yfoject 
was meeting its goals. 

Methodology . The field test was conducted wiili zix teachers and approximately 120 
students who were predominantly Mexican American and economically disadvantaged. At 
the beginning and end of the field test period (the first semester of the 1971-72 school year), 
students completed a short questionnaire designed to measure their: (1) preferences for 
school subjects, (2) self-concept of school ability, and (3) attitude toward school. Periodic 



"^Education Service Center Region 1. CVAE (Coordinated VocationalAcademic Educa- 
tional) Academic Curriculum Project Evaluation Report, 1971-72, (Edinburg, TX: Education 
Service Center Region 1, 1972). 



classroom observations were made to determine the degree of implementation of the curricula 
and the type and extent of student activities. Data such as school attendance and student 
discipline were obtained from scliool records. Curriculum imbedded criterion-referenced tests 
were used as a measure of student achievement. 

Results, Students* attitudes toward school and self-concept of school ability showed 
no statistically significant gains. A significant number of ninth grade students changed their 
opinion of science from ^Mislike" to **like/' Classroom observation data showed that the 
curriculum was individualized and students spent a large portion of their time in individual 
activities. The rate of abs^ieeism dropped from 8.9% to 7.9% and the rate of CVAE disci^ 
plinary referrals dropped /rom 28.4% to 11,7%, **Studcnts demonstrated mastery of specific 
performance objectives, from 88.7% to 99,8% in the different curriculum areas. More signif* 
icantly* they demonstrated short-term retention rates which varied from 33.2% to 77.4%" 
(Education Service Center, 1972). 

Evaluation Model for Career Programs 

After developing an innovative curriculum, a district is often left with the problem of 
how to validate the appropriateness of the curriculum. Here is how one local agency handled 
that challenge. 5 

Project Overview , This project was designed to develop and test a model for qualitatively 
evaluating career programs. The results of the evaluation specify curricular areas possibly in 
need of revision. 

Purpose of the Evaluation , To determine the effectiveness of the evaluation model under 
actual conditions. 

Methodology . The two career training areas chosen for tHis study were (1) dental assis- 
tant, and (2) auto mechanics. The curriculum in each of these areas was written in a format 
which included precise behavioral objectives. To determine the appropriateness of the cur- 
riculum in each training area, a population of interested and involved judges was identified. 
This population consisted of the following six groups: (1) students, (2) instructors, (3) 
advisory board, (4) graduates of the two training areas, (5) employers of graduates from the 



^Richard L, Byerly, et al.. Evaluation Model for Career ProgramSy Final Report {Ankeny, 
lA; Des Moines Area Community College, 1972). 
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two training areas, and (6) businessmen who have not yet> but may in the future> hire graduates 
of the two training areas. A population of the above mentioned judges was identified for 
each of the training areas. The judges were asked to rate each behavioral objective in their 
training area using the following scale; 

0 - of no importance 

1 - desirable but not necessary 

2 - desirable and necessary 

3 - absolutely essential 

Each of the training areas was divided into subareas. Group respondent scores were 
combined for each subarea and a mean was computed. An analysis of variance was then con- 
ducted to determine if any significant differences existed. If significant differences were 
found, paired "t** tests were used in an attempt to determine which groups were responding 
differently. 

Results. Several sub-sections in each of the training areas were found to contain signif- 
icant differences. For example^ in the electrical subarca of auto mechanics, the analysis of 
variance was significant at the .05 level. The paired "t" tests indicaced that the employers, 
students and ex-students were placing less emphasis on these items than the other three groups. 

Using this approach, there were several areas whert- significant differences did 
exist between groups. These differences illustrate that the six groups perceive 
the tasks at different stratas of importance. The implication, tneretore» is that 
curricular revision may be necessary in these areas where divergence was apparent, 
If revision is not necessary^ at least closer examination of the curricular ditier- 
ences could be accomplisned (Byerly, et al., 1972:40). 

Project VIGOR 

This study made extensive use of the written questionnaire to obtain information. It 
also has the rather unusual characteristic of having a student conduct a portion of the eval- 
uation.^ 



^David Douglas Public Schools, Project VIGOR: Vocatioml Cluster Education, Inte- 
grated and Articulated Grades 1 Through 14 with Guidance Services, Occupational Exploration 
and Work Experience Relevant to General Education, Final Evaluation* (Portland^ OR: David 
Douglas Public Schools, 1972). 



8 



Project Overview ^ Project goals extend from career awareness at the primary grades to 
student follow-up and placement at the post high school level. 

Through Protect VIGOR, the David Douglas Public School System (Portland, 
Oregon) is addressing itself to the objective of changing a conventional academ- 
ically*oriented, general education school system into one whose curriculum 
reflects the needs of all students regardless of entry into their chosen vocation 
(David Douglas Public Schools, 1972). 

Purpose of the Evaluation . 

The evaluation was purposely designed as a very subjective, human oriented 
evaluation strategy. The overall purpose was to gather as much information 
as possible about current teacher and administrator knowledge, feelings, and 
attitudes about project status for developing base line information for more 
sophisticated future evaluation (David Douglas Public Schools, 1972:9). 

Methodology. Eight separate questionnaires were developed to ellicit responses relevant 
to project goals and/or objectives, These questionnaires were given to two different groups- 
administrators and teacher-counselors-at four different levels: elementary (grades 1-6), 
middle school (grades 7-8), junior high (grades 9-10), and senior high (grades 11-12). The 
response ratio was 60% from teacher-counselors and 85% from administrators. To verify 
responses obtained from the questionnaires, an interview was conducted with 33 David Douglas 
teachers, counselors and administrators. In addition to the questionnaires and interview, 
student information was collected by a member of the senior class at David Douglas High 
School. The student information was relevant to the knowledge and attitude of ninth grade 
students toward career education. 

Results. The findings and conclusions reported were substantial; those of general interest 
include the following. Awareness programs at grades 1-6 may become the unifying thrust 
needed for other components. However, "two apparent inhibitors of program changes within 
the institution: (1) the contention that this isn't really new, just a different language; and 
(2) the conviction that 'this too shall pass away if we all just ignore it' " must be reconciled 
(David Douglas Public Schools, 1972). 

The evaluator recommended that these inhibitions "might be overcome by evidence of 
positive results from the career education program" (1972). It was interesting that most 
administrators appeared to understand the thrust of the project while few teachers did. While 
involvement of employers and community groups was successful, there were limitations on 
the range of experiences which could be provided students due to the physical makeup of 
the local community. Considerable more effort is needed to rncrease ninth grade students' 
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knowledge of the career education concept and its relation to subject areas. To allow for a 
more comprehensive evaluation of the project, *Mndicators of performance and/or accomplish- 
ment need to be detailed for each project component" (David Douglas Public Schools, 1972)« 

Central Kentucky Vocational Education 
Evaluation Project 

This study had several purposes including the development of a formal model for eval- 
uation of vocational education. ' 

Project Overview ^ The Central Kentucky Vocational Education Evaluation Project 
(VEEP) was a pilot project designed to develop expertise at the appropriate level for organiz- 
ing and implementing an effective evaluation of vocational programs. Its objectives were to: 

1} identify new or improved procedures for assisting schools in conducting program 
evaluation, 

2) test and demonstrate evaluation procedures to determine whether the secondary 
and post-secondary programs of vocational education in Central Kentucky are 
fulfilling the stated objectives, 

3) develop state and local leadership competencies needed for evaluating programs 
of vocational education, and 

4) enable personnel in the local and regional schools to be more proficient in the 
various tasks necessary for an adequate evaluation. 

The project focused on product evaluation which emphasized students' attainment of criterion 
behavior as stated in objectives, The project activities consisted of conducting conferences, 
maintaining consultive services, developing an evaluation manual, analyzing school data, and 
preparing project reports. 

Purpose of the Evaluation ^ The project was evaluated to determine the extent to which 
it met its stated objectives. 



'Kentucky Research Coordinating Unit, Central Kentucky Vocational Education Eval- 
uation Project, Final Report (Lexington: Kentucky Research Coordinating Unit, October, 
1972), 
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MethodoloRY* Seminars conducted by the VEEP staff were evaluated by participants 
responding to questionnaires. VEEP staff members made periodic visitations to participating 
local school districts to observe the evaluation efforts and to interview local leadership teams. 
Each participating school district was asked to prepare quarterly reports of their activities 
which supplied further evidence of local involvement. At the conclusion of the project, a 
follow-up questionnaire was sent to all participants to determine their perceptions of project 
effectiveness* 

Results^ A formal model for evaluation of vocational education was adapted from several 
sources including those developed in Michigan (Byram, 1965). An instrument to obtain follow- 
up information from former vocational education students was developed and utilized by 12 
participating school districts. Several workshops were conducted for school district staff 
members to assist them in acquiring the abilities and knowledges necessary to perform an 
adequate program evaluation and develop measurable performance objectives. The project 
also developed and maintained effective public relations through the use of various media, 
including a newsletter. 

SOME CONSIDERATIONS FOR 
PROGRAM EVALUATION 

For discussion purposes, the planning and evaluation system used in the VEEP project 
(Kentucky Research Coordinating Unit, 1972) will be used as a guide when discussing some 
of the things one should consider when evaluating a vocational-technical education program 
(see Figure 1), 

The reader should understand that this is only one example of a system or model for . 
evaluation. Many different systems exist, varying in complexity, design and intent. For 
examples of other systems, the reader is referred to Provus (1969), Stufflebeam (1971)^ EPIC 
Evaluation Center (n.d.), and Stake (1967)« This is not an exhaustive listing of evaluation 
systems, but it does provide for a wide variance in methodological approaches. 

There are at least two important considerations when choosing a system for evaluation: 

1) Is the system appropriate for the situation to be evaluated? 

2) Are the evaluator and others concerned capable of handling the complexities of 
the system? 
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FIGURE 1. -PLANNING AN EVALUATION SYSTEM 
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It is uncomfortable and counter-productive to be saddled with an evaluation system too com- 
plex for some of the partixipants. As Majer (1972) points out when referring to a situation 
involving the CIPP model: 

One of the basic, pervading problems was the use of a moderately complex model 
for evaluation, the CIPP model. This is no indictment of the model. It is based on 
sound theory and can be very useful in program evaluation. The problem rather 
was a result of the model being somewhat complex to use with participants (staff, 
graduate students^ and school personnel) whose professional expertise is in other 
areas. It is not a model for people untrained in evaluation. One bit of evidence < 
for this was the continued confused discussion (and resulting frustration) about 
whether a given activity was **context** or 'input" evaluation. In reality this is not 
an important question. The important question is whether one can develop a strat- 
egy for getting valid information necessary to make decisions 

Majer suggests choosing a rather straightforward system for evaluation and examining some 
of the considerations along the way through the system. 

Assess Needs 

Kaufman contends that, 

the identification of needs is a discrepancy analysis that identifies the two polar 
positions of: 

Where are we now? 
Where are we to be? 

; and thus specifies the measurable discrepancy (or distance) between these two 
poles (1972:29). 

The actual assessment of needs can be accomplished in any number of ways from a 
community analysis (Cromer, 1968), to a determination by authorities. Regardless of how 
needs are assessed, it is important to the planning process that the information gained is valid. 
From the discrepancy identified through the needs assessment, an idea is formulated to, hope- 
fully, remove the discrepancy. 

Etevetop Philosophy 

Before a remedial program (to overcome the identified discrepancy) can be developed, a 
basic guiding philosophy should be developed. One of the concerns for such a philosophy 
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should be: "Whether or not the remediation can be effected entirely through the effort of 
the local agency or should outside assistance be recruited?" Perhaps the decision is made to 
adapt some innovation developed elsewhere to fit the local situation. In such a case^ an 
evaluation scheme such as that reported by Hull and Wells (1972) would be appropriate to 
assist in choosing the correct innovation. Whatever philosophy is chosen for guiding the 
development of the remediation, it should not be viewed as inviolate, but changeable if evi- 
dence is presented indicating a need for change. 

Write Objectives 

Now that a need has been determined and a guiding philosophy developed, program 
objectives should be identified. These may be written, as in the case of a locally developed 
program, or adapted from an existing program. Regardless of the source, as Bloom (1971) 
points out, to facilitate the evaluation process, objectives must be "meaningful, unambiguous 
statements of intended educational outcomes/' Often objectives are written that only identify 
inputs rather than outcomes. An example might be: 

"Staff development will be provided for teachers.** 

or 

"The students will view a 30 minute film/' 

This type of an objective can be evaluated quite easily by verifying the input, but unless some 
outcome Is specified, the evaluation cannot say whether or not any desired results occurred. 
The program objectives will determine to a large extent the evaluation that is conducted; 
therefore, time and energy spent in their careful construction will bring many benefits. 

If the program is of an instructional nature, then the research reported to date suggests 
that behavioral objectives are probably beneficial (Walbesser, 1972). There seems to be a 
positive effect from having the learner informed of the expected learning outcomes, and 
learner knowledge of the objectives seems to increase the rate of acquisition as well as decrease 
the rate of forgetting, A number of publications are available to assist in developing behavioral 
objectives, including Popham (1970)> McAshan (1970) and Walbesser (1970). Critics of 
behavioral objectives (Eisner, 1972) generally see them as producing too narrow a view of 
education. 

State Criterion Questions 

An evaluator has been characterized as an extension of the mind of the decision-maker 
(Stufflebeam, 1971). His role has two major components-the technical and the interface. 
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It is within this interface role that the evaluator must couch the evaluation's criterion ques- 
tions. Scriven (1967) has typilled evaluation as attempting to answer certain types of ques- 
tions about various educational processes, programs, and the like. 

In the evaluation system used in this paper, the evaluator, within his interface role and 
by paying attention to the program objectives, can develop a meaningful list of criterion 
questions. Welty (1970) describes the painstaking steps necessary in the development of a 
series of questions to formulate an evaluation design. He advocates that the evaluator meet 
personally with people interested in the program so that they will have input into the develop- 
ment of the evaluation design. 

It is the opinion of this author that a formal document should be made of the criterion 
questions, along with all appropriate information relative to what data will be collected for 
each question, by whom it will be collected, and when it will be collected. This document 
should then be approved by the appropriate authority, i.e., in the case of a local school 
district, the school board. For an example of an evaluation design, the reader is referred to 
Fielstra(1971). 



Collect Data 

The collection of data and its subsequent treatment can be classified generally into two 
major activities. The first is the monitoring of the project as it is implemented. The second 
is measurement of attainment before, after and as many times as necessary during the project. 
These two activities have been variously defined and named in the literature. Scriven (1967) 
and Bloom (1971) refer to them as formative and summative evaluation. Stufflebeam (1971) 
and others refer to them as process and product evaluation. For the remainder of this paper, 
the latter classification scheme will be used. Process evaluation is explained as providing 

. . . periodic feedback to persons responsible for implementing plans and proce- 
dures. It has three objectives: (1) to detect or preoict defects in the procedural 
design or its implementation during the implementation stages (2) to provide 
information for programmed decisions and (3) to maintain a record of procedure 
as it occurs (1971:229). 

Product evaluation **measures and interprets attainments at the end of the project cycle and 
as often as necessary during the project term" (Stufflebeam, 1971:232). 

Certainly both process and product evaluation are needed. As Glaser (1972) said in 
discussing his model which he calls the ^'clinical approach": 
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assessments of research or service efforts will be most meaningful and useful 
if they are designed to evaluate the program as a whole and as a dynamic, 
complexly interrelated entity. 

If a program should be assessed by product evaluation only^ a great deal of risk exists in not 
knowing what the program really was. 

Norton was certainly correct when he said» **to evaluate process is much easier said than 
done** (1972:62). However, authors such as Eisner (1972) make a strong case for the careful 
observation and description of the program environment. It has been the observation of this 
author that careful attention to process evaluation has at le^st two major redeeming qualities 
which make it worth the large expenditure of resources. First, it demands that the evaluator 
become welt informed with the functioning of the program and the complex interrelatedness 
of the many variables involved. Second, information provided from periodic observations 
serves to alert management to any design-implementation discrepancies, 

Product evaluation needs little if any justification, since it is aimed at the measurement 
of program attainments. However, a thorough consideration for the many data sources for 
product evaluation is necessary for a comprehensive evaluation (Webster, 1973). 

There are many types of data that can be collected and different methods of collecting 
them. Some of the more common ones are discussed below with additional sources of infor- 
mation included where appropriate. 

One of the more obvious methods of data collection is researching the school files and 
records. These files include such items as student cumulative records, and personnel, business, 
and attendance records. As with other data» these data should be handled in such a way so 
as to insure individual anonymity. Some of the types of data which can be obtained by 
searching existing records are: 

1) student demographic data including birthdate, grade point average, attendance, 
address, and test scores; 

2) teacher demographic data including years of experience, major and minor field, 
and degrees held; and 

3) school demographic data including enrollment by grade, by section, average 
daily attendance, and schedule of classes (Webster, 1973). 
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Individual and group testing is one of the most widely used methods for obtaining eval- 
uative data. There are a number of sources available to the reader which discuss the various 
tests that are presently available. Probably the most well known is the Mental Measurements 
Yearbook series (Buros, 1972). 

Findley (1971) has an important discussion of factors that affect test results. He points 
out the importance of test administration, scoring and recording as they effect the results. 
It is important that the evaluator monitor at least a sample of each test administration to 
identify any possible problems. Clemans (1971) presents a list of general rec6mmendations 
for proper planning of test administration which should assure smooth testing procedures. 

As Seibel (1968) points out, achievement tests, scholastic aptitude tests and intelligence 
tests are quite similar. 

In fact, if we were to look at an item from a test without knowing its source, 
it would be difficult and perhaps impossible to specify with assurance whether 
the item came from a test that is labeled as an achievement test, a scholastic 
aptitude test, or an intelligence test (1968:265). 

Of the three groups of tests, perhaps the type of test most often used for program evaluation 
is the achievement test (Downie, 1967). 

A second group of tests are those classified generally as attitude-interest tests. Partic- 
ularly the vocational interest tests have proven effective to counselors. Downie (1967) 
identifies the problems with attitude tests as: 

1) Low correlation between measured attitudes and observed behaviors; 

2) Low consistency between specific items and general concepts; 

3) Change in attitudes over time; and 

4) The interpretation of neutral attitude. 

He also identifies some problems with vocational interest inventories as: 

1) They can be faked. 

2) They have a high level of reading difficulty. 



3) 



The possibility exists of always receiving socially acceptable responses (Downie, 



A third classification of tests are criterion-referenced tests, Brazziel (1972) explains 
criterion-referenced <'csts as measures of **student progress toward explicit objectives as 
defined by the school enterprise. They are measures of degree of mastery of material taught 
and learned in a specific time frame." Thus, from a student^s incorrect response to a criterion- 
referenced item or set of items, a teacher can prescribe further remediation for the student by 
referring back to the antecedent objective, A number of test publishers are currently publish- 
ing and marketing criterion-referenced tests. 

Questionnaires and interviews are also widely used methods of collecting data for eval- 
uation purposes. Phillips lists the relative ease with which an evaluator can obtain a great 
deal of information from a probability sample as one of the virtues of these methods {Phillips, 
1966). Questionnaires and interview schedules can vary from open-ended to closed-form 
instruments. The closed-form instrument is advised when categorized data is needed^ whereas 
the open-ended instrument is best suited for preliminary exploration of untried situations. 

The primary considerations in developing questionnaires and interview schedules are; 
(1) keep the questions simple and concrete, (2) questions should not be open to subjective 
evaluation, and (3) do not bias the response (Young, 1966). Young contends that a carefully 
worded questionnaire provides privacy and anonymity while an interview conducted by a 
well trained interviewer is a highly flexible tool. Either method, of course, should be pilot 
tested before using. 

The author has found that interviewing a sample of respondents and, if possible, non- 
respondents to a questionnaire survey is well worth the extra effort. Verification of how 
subjects are interpreting questions can be obtained by interviewing respondents and an indica- 
tion of bias introduced from not having a 100 percent retum can be determined by inter- 
viewing non-respondents. 

Direct observations are another method of data collecrion. Although this method is very 
time consuming and thus costly, the author has found the information gained well worth it. 

Roberson (1970) presents six different observational categories. This discussion will be 
limited to on-site activity observations or **what people do.*' He gives several recommenda- 
tions for selecting or developing an observation system. 

1) The observation system should include terms, 

2} The terms should be simply defined. 



3) 



Examples of the activities to be observed should be given. 



4) The observer should be provided an easy format for coding observations (1970), 

Phillips (1966) recommends combining the interview technique with observation. The 
two methods complement each other since interview data provide what the subject says he 
does while observation data provide information about what he actually does. 

From the experience of the author, it has been found that when observing classrooms, 
teachers appreciate knowing what the observer is doing. Results of observations can be dis- 
cussed with teachers; and if observations are to be maintained over a period of time, better 
rapport will result. It is realized that there is a danger of biasing the situation by telling the 
teacher what is being observed^ but experience has shown that when classroom activities are 
being periodically observed, teachers tend to ignore the observer. 

Another method of data collection is through the use of unobtrusive measures {Webb, 
1966). This type of measurement is difficult to design. An example is measuring the extent 
that books are read by judging the amount of wear and tear on the book itself. 

Regardless of the method used for collecting evaluative data, there are different means 
of collecting it. Teachers can collect it (Education Service Center, Region 1, 1972); students 
can collect it, and internal evaluators can collect it (David Douglas Public Schools, 1972); 
and the community can provide it (Byerly, 1972). 

Two separate types of vocational education program evaluation require a little different 
collection of data or possibly a different use of data. Follow-up studies are designed to deter- 
mine what former students are doing after leaving the program and how the program effec- 
tiveness is perceived. Voelkner (1971) lists the variables of interest in a follow-up study as: 

1) length of time a graduate takes to find a satisfactory job, 

2) employment security, 

3) length of time on the job, / 
. 4) earnings progression, and 

5) rate of advancement. 

Wilson (1971) sees the follow-up of students as a way of increasing vocational education 
accountability with the community by making business and industry more aware of the pro- 
grams, Follow-up studies also provide information relative to program changes (Jacobson, 
1971; Bison, 1972), 
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Cost-efrcctiveiiess studies are an attempt to attach costs to program inputs and/or out- 
puts. Kaufman (1969) provides an introductory discussion of the subject. Connor (1972) 
purports to shows how cost-benefit analysis can be used to determine the most effective way 
of combining vocational education with general education. Stanton (1970) shows how the 
costs of vocational education are far less than for manpower programs. It is the opinion of 
the present author that trying to attach costs to program inputs can be accomplished with 
reasonable satisfaction. However, trying to attach costs to program outputs is very difficult 
because of problems of definition and measurement. 



Analyze Data 

There are a host of books related to methods of statistical data analysis. Two excellent 
general texts are by Hayes (1972) and Glass (1972). There are also a number of books that 
discuss the use of computers in analyzing statistical data. An introductory one is by Veldman 
(1967) while Cooley (1971) has a more advanced one. The author will not attempt to discuss 
the many ways available to analyze data; however the evaluator should ascertain that the 
analysis is appropiate to the data collected. As a matter of routine, all computer programs 
should be tested on a set of sample data before being used on actual data. 

Since the present system for evaluation being used for this paper does not indicate where 
reporting of data should appear^ the author has chosen to insert it at this point* I; has been 
the experience of the author that unless the report is written in a brief, easy to understand 
manner it will turn more people off than on. If a sophisticated statistical analysis is needed, 
by all means than the evaluator should do it; but when the report is written, the analysis 
should not so confuse the reader that the results are lost. An "executive summary'^ similar 
in form to that used in th^ second section of this paper should precede any report* This will 
facilitate understanding on the part of the reader. 



Formulate Recommendations 

The evaluator is obligated to provide any information suggested by the data as possible 
methods for improving the program (Cameron, 1971). This author concurs; however, the 
evaluator must be cautious and recommend only that which is suggested by the data. The 
recommendations should appear in both the executive summary and in the text of the report. 



Make Decisions 



Hopefully, through the use of process evaluation, information has been provided to 
management since the inception of the program and many decisions have been made 
(Stufflebeam, 1971). The evaluator should not be surprised If a recommendation is not 
acted upon; there is a political dimension to the decision-making process. 

Decisions can be made to start the evaluation cycle all over again by reassessing needs 
or go to some other phase of the cycle. 

SOME FtmTHER CONSIDERATIONS 
ABOUT PROGRAM EVALUATION 

There are two aspects of the evaluation of the Skyline Career Development Center in 
Dallas, Texas that seem to extend beyond anything currently found in the literature. The 
Career Development Center is a large complex housing 28 career clusters. Any senior high 
student in the Dallas Independent School District is eligible to apply for enrollment in any 
of the career areas. Transportation from each of the local high schools to the Career Develop- 
ment Center is provided. The clusters are organized in a three hour instructional block of 
time so that a student is only in a given cluster for either the morning or afternoon session. 
The instructional staff consists of 103 instructors for a student population of about 2,350. 

Context Evaluation 

In an attempt to get an accurate profile of the Career Development Center student pop- 
ulation, a great deal of information oh each student is obtained and recorded in computer file 
The following is a list of all data sought for each student. 

1) Identification number 

2) . Home school 

3) Birth date 

4) Parent name 



5) 



Address 



6) 



Phone number 



7) 


Sex 


8) 


Ethnic background 


9) 


Grade level 


10) 


Cluster-(course) 


11) 


Period of day (A.M. or P.M.) 


12) 


Parent occupation 


13) 


Mode of transportation to Skyline 


14) 


Scores from 9th grade Iowa Tests of Educational Development 


15) 


Scores from 8th grade California Test of Mental Maturity 


16) 


Social Security Number 


17) 


Previous Career Development Center attendance 


18) 


Scores from Armed Services Vocational Aptitude Battery 



The information within the data base is updated as changes occur. Computer generated 
analyses of the student data are produced in the fall, at the end of the first semester, and 
again at the end of the school year* With the computerized data base it is relatively easy to 
generate a random sample of students, report periodically on the withdrawal rate, and provide 
instructors and administors with different types of information needed for annual reports. 

Upon graduating from the Career Development Center, a student's record is moved to 
the graduate file which is used to conduct an extensive follow-up evaluation. Currently this 
information is being used to research cost-effectiveness as it relates to obtaining follow-up 
information. 



Process Evaluation 



In order to provide information relative to program implementation, classroom activities 
are regularly monitored by trained observers. Three rounds of observations are scheduled 
each year* A round of observation is defined as the length of time necessary to observe all 
clusters. For each round of observations a different set of factors are observed* The design 
for process evaluation is graphically represented ux Figure 2. 




Sep Oct Nov Dec Jan Feb Mar Apr 
DURATION OF PROJECT 



FIGURE 2.-NUMBER OF OBSERVATION vs LENGTH 
OF OBSERVATION PER ROUND OF OBSERVATION 



At the beginning of the project year, four observations of approximately 20 minutes 
each are made in each cluster. For this first round of observation, the factors being observed 
include such things as adequacy of facilities and materials. During this time, one short inter- 
view with each instructor is conducted. By the end of the year, the observations consist of 
one observation of about three hours duration for each cluster* During this time, students 
are interviewed to determine their attitude and perceptions. The reader must realize that 
Figure 2 is only a **road map" and in actual practice the number and length of observations 
will vary to fit the situation. The overall design seems to work well since the frequent short 
visits initially help the evaluator develop rapport with the situation and the various instructors. 
By the end of the year, the student information gained from the interviews is very valuable. 
Reports on process evaluation information are provided to management about every two 
weeks or soOner> depending on the frequency of visits. 



EPILOGUE 



Kindschy (1971) makes a strong point for the evaluation of the vocational education 
department by advisory councils. The author sees a great deal of value in using the business 
and industrial community as sources of information. What better judges can be chosen for 
program evaluation, particularly curriculum validationi than the fina! users of the product 
of vocational educationi the employers. This type of validation, coupled with hard data to 
verify student and staff satisfaction, student achievement, and cost-effectiveness^ will com- 
prise a comprehensive evaluation. 

of course evaluation costs money, and the plea for funds has gone out (National Advi- 
sory Council on Vocational Education, 1971). if a justification for further expenditures is 
needed, then heed what Armstrong (1969:16) says: 

A primary defense against an undesirable reaction to the many good programs 
in Vocational Education is careful evaluation of results to insure the elimination 
of defective ones before they create a public scandal. 
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